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ABSTRACT 



This booklet offers strategies for leading small groups as 
they investigate the status of school reform. The tools described were used 
in the collection of information from focus groups in New York, 

Massachusetts, and Vermont. Some of the groups were comprised of school 
practitioners, while others were made up of providers of support to schools, 
such as professional developers and professors in teacher-preparation 
colleges. A conscious effort was made to include people who occupy different 
places in the systems that link state policy to the classroom. Members of 
each group were brought together to reflect on the reforms taking place and 
on the systems that support or could support those reforms. To facilitate 
these meetings, groups of 8-15 people were asked to focus quickly and 
thoughtfully on each of 3 areas. Some of the topics include the roles assumed 
by the state department of education in leading reform, philosophies of 
reform and change reflected in state education department strategies, state 
department of education relationships with other facets of the education 
system, and the reception of reform initiatives in the field. The tools used 
for each group are outlined and include information about their use, key 
features, and suggestions for adaptation. (RJM) 
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What is in this document? 



This paper includes a set of tools designed for leading small groups in investigating 
the status of school reform. These tools were developed as part of a study by the 
Northeast and Islands Regional Educational Laboratory at Brown University (LAB) 
of the changing roles of state departments of education. 

This study involved collecting information from focus groups in several states. Some 
of the groups were comprised of school practitioners, while others were made up of 
providers of supports to schools (professional developers, professors in teacher 
preparation colleges, etc.). Members of each group were brought together one time to 
reflect on the reforms taking place in the schools they know and on the systems that 
support or could support those reforms. To facilitate these meetings, it was necessary 
to help groups of eight to fifteen people — some of whom did not know each other — to 
focus quickly and thoughtfully on each of three areas under discussion. The questions 
and activities used to provide this focus are the subject of this paper. 

For applied research that requires reflection by informants on complex issues, focus 
groups represent an increasingly useful approach. The tools provided here have been 
successfully adapted to discussions of varied but specific reform initiatives in very 
different state environments. They have been used in urban, suburban and rural areas. 
At focus groups in which variations of these processes have been used, participants have 
reported that they found the conversations informative, provocative, and enjoyable. 
Some participants who use focus groups in their work have asked whether they could 
adapt the tools and processes for their own use. 

Based on such response from participants and on the richness of the data we collected 
for the LAB's study, we believe the format of these tools will be of interest to other 
researchers. 




IV 







0 



tudy of State Strategies 




LAB Study of State Strategies 

When the leadership team of the LAB met in the spring of 1996 with education 
leaders in each state and territory of the region, one of the research priorities identified 
was related to the changing roles of state education departments. 

State leaders explained that their departments of education have for several years been 
in a period of vital transition, with their reforms going well beyond procedural changes 
and adoptions of new programs and curricula. The reforms of both the standards 
movement and the federal legislation of the 1990s have called for complex and 
comprehensive changes in every school and classroom. At the same time, state 
departments of education have been changing. Most are now smaller than they were 
ten years ago, and they have made concerted efforts to move their relationships with 
schools away from regulation and monitoring and toward vision-setting, support, and 
capacity-building. Several of the region’s state-level leaders asked the LAB to help 
them study new leadership and support roles adopted by state departments of 
education, including the role of commissioners. The goal was to determine whether 
role changes have proved sufficient and effective for scaling up deep reform in today's 
standards-based education environment. 

Working in the fall of 1996 with commissioners' representatives from Massachusetts, 
New York and Vermont, we identified a framework of issues that commissioners and 
state departments of education would like to better understand as a result of taking 
part in this study. The full framework of questions they raised is available elsewhere. 
The areas of inquiry are summed up as follows: 

■ Roles assumed by the state department of education in leading reform 

These include responding to demand or creating new directions, disseminating 
well developed programs and procedures or initiating local design efforts, and 
working directly with schools or through intermediate agents. 

■ Philosophies of reform and change reflected in state education department strategies 

These can be related to the level of the school system targeted for information, 
flexibility regarding local implementation, and connections drawn between 
initiatives. 

■ State department of education relationships with other facets of the education system 

These include linkages between state initiatives and the work of universities, unions, 
or professional organizations, as well as new networks or structures created to 
support reforms. 

■ Reception of reform initiatives in the field 

This includes differences in the ability of various kinds of schools or categories of 
teachers to implement specific reforms, as well as local ownership of the goals of 
the reforms. 

Based on our inquiry into these areas, the LAB is developing principles and guidance 
for use by state-level policymakers across the region. 
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Data collection 



In order to answer the questions about the roles and strategies adopted by state 
education departments, it was important to view the issues from multiple perspectives. 
For example, by working with more than one state, we could see a variety of strategies 
across different educational and organizational cultures. For that reason, we chose to 
work with Connecticut, New York and Vermont. 

In each state we spoke with people who occupy different places in the systems that link 
state policy to the classroom. This enabled us to hear how widely approaches from 
state departments of education have reached, how clearly intentions have been 
transmitted, and by what means this occurred. Because these questions required deep 
discussion of complex circumstances, we used focus groups for some of our data 
collection. In each state we met with groups at three levels of the reform system: 

Practitioners 

Classroom teachers, building administrators and district administrators 
Support system 

Staff developers, teacher network leaders, managers of agencies established to 
support specific programs or initiatives, professors from teacher preparation 
institutions, trainers from professional organizations 

State policymakers 

Commissioner, senior state department of education staff, state board of 
education representative, key legislator 

At the practitioner and support system levels, we conducted multiple focus groups in 
each state. This made it possible to hear from people in urban, suburban, and rural 
settings; from different areas of the state; and from both resource-rich and resource- 
poor school districts. 
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About focus groups 

The focus group is a specific group interview technique. Groups are generally small 
(six to twenty-two participants in our study), and participants are selected to be 
representative of a broader population. In each of our New York and Vermont 
practitioner focus groups, for instance, we had teachers and administrators from 
different subject areas and grade levels, staff of high-achieving schools and low- 
achieving schools, and people with different degrees of familiarity with specific 
support strategies being used by the state department of education. A focus group is 
tightly facilitated and is structured around a set of questions. The purpose of the 
meeting is to elicit from the participants detailed responses to these questions. 

Focus groups are an increasingly valued part of a repertoire of tools for applied research 
and have specific advantages. First, they allow the researcher to investigate the 
complexities and individual experiences behind the kind of initial answer one might 
get on a survey. They allow informants, or subjects of research, to participate in and 
shape the analysis of findings. And, in a way that individual interviews cannot, they 
allow informants to learn from and influence each other and to elaborate their ideas 
based on the contributions of others. 

Still, these advantages are accompanied by limitations. One is that a focus group is not 
interchangeable with a survey. The group, while representative, is always small and is 
rarely the kind of structured sample from which one would draw statistical inferences. 

So focus groups are used to illuminate a complex and/or subjective issue rather than to 
describe an entire population. Also, because the data derived from a focus group come 
out of deep conversation, the number of questions must be kept small. In short, these 
groups provide depth rather than breadth, which places a responsibility on their 
designers and facilitators to know enough about an issue in advance to ask important 
questions that do not have simple answers. Also required is a structure that brings a 
group together quickly (focuses it) so that all participants are addressing the same 
issue, albeit from their different perspectives. 

The tools that follow are examples of the ones LAB staff developed, in consultation with 
two state departments of education, for the range of groups described earlier. We have 
adapted these formats to different content in each state. Across these contexts, we have 
found that such tools accomplish the following: allow a group to exchange more 
information than there is time to discuss, focus conversation quickly on important and 
complex questions, place value on each participant's experience, and provoke lively 
conversation. And, critical to a focus group, they do this without a great deal of 
explanation by the facilitator, leaving the bulk of the meeting time for participants to talk. 

Participants in each of the focus groups we have conducted for this study have said 
they enjoyed and learned from the conversation. We hope that other researchers 
investigating other questions will find these formats equally effective. 
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Agenda for a practitioner focus group 

Four of the following five instruments for focusing conversation and collecting 
additional information were used in one series of focus groups in New York state. 
The participants in these groups were practitioners: teachers and administrators. 

The study in New York did not focus on any single initiative, but rather on all the 
efforts of the New York State Education Department to support the implementation 
of challenging standards in all the state’s schools. 

The following outline describes time allocations and materials for these three-hour 
focus groups. Each of items B - D represents a separate conversation, with its own 
initial activity to focus discussion. The tools used in these initial activities and in 
agenda item E can be found on pages 7-15 of this paper, along with suggestions for 
their use and adaptation. 



Activity 


Minutes 


Materials 


A Welcome and introductions; 
overview of purpose of study, 
purpose of focus group, and 
organization of the meeting 


15 


✓ Name tags 
S Participant list 

S Background information about the 
study and the LAB for participants 
to take home 


B Understandings and 
implementation of reforms: 
questionnaire and discussion 


45 


S Two-page survey (pp. 7-8) 
•S Blue and yellow index cards 
✓ Card display stand 


C State roles and strategies in 
scaling up reform: 
dot voting and discussion 


60, 

including 
break * 


✓ One-page summary of eight 
strategy scales (pg. 10) 

✓ Eight wall posters with one scale 
each 

✓ Adhesive dots 


D Support systems for reform: 
questionnaire and discussion 


45 


✓ Two-page survey (pp. 12-13) 
S Blue and yellow index cards 
S Card display stand 


E "The Last Word": 
questionnaire 


10 


S One-page survey (pp. 15) 


F Wrap-up 


5 





* Because these practitioner focus groups were held from 4 to 8 p.m., the time most 
convenient for the practitioners, we served a light dinner at this point in the evening. 
For meetings at other times of day, with a more conventional 15-minute break, this 
agenda item was completed in 45 minutes and the 15 minutes saved was redistributed. 
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^focus Group Tools 



Focus Group Tools 

The paper tools used in the New York practitioner focus groups follow on pages 7-15. 
For each we provide information about its use, its key features, and suggestions for 
adaptation. One such adaptation is found on pages 17-19, in a tool used with support 
system focus groups in Vermont. 

Understandings and implementation of reforms: 
practitioner questionnaire 

This two-page questionnaire is designed to elicit two kinds of responses. The first page 
provides the focus for discussion. The second page asks for information about which 
the New York State Education Department was interested regarding the status of 
several specific initiatives; this information was collected for the LAB study but not 
discussed in the focus group. The full instrument is completed in 5 minutes, leaving 40 
minutes for the discussion that is the data of focus groups. The questionnaire is 
collected before the group adjourns. 

The first page consists of a series of matched statements. Each participant works alone 
to read these statements and check the one in each pair — either the statement in the 
blue column or the statement in the yellow column — with which he or she most 
nearly agrees. (When the questionnaires are used, the "blue" heading is actually colored 
blue and the "yellow” heading is colored yellow.) Each pair of statements has the 
following characteristics: 

■ The statements are unambivalent and for all practical purposes opposite, forcing 
the respondent to "take a stand," or identify his or her opinion. 

■ The statements are personal opinions based on experience, establishing 
immediately that subjective experience is the basis of the discussion. 

■ The statements are descriptive rather than judgmental, so that a participant need 
not feel defensive about his or her choice. 

■ The more "desirable" statements, the ones that suggest further progress in 
implementing reform or closer agreement with state policy, do not always appear 
in the same column, so that neither blue nor yellow becomes identified with the 
right answer. 

The blue and yellow headings facilitate the sharing of responses across the group. Each 
participant has at his or her seat one blue index card, one yellow index card, and a 
holder to display one card. (Holders available commercially to display name cards 
work very well, or the facilitator can make them by folding card stock into a "tent" 
with a pocket or "ledge" on the front side.) Displaying responses serves multiple 
purposes. First, there is not time in the 40 minutes provided to discuss all the questions 
on this questionnaire; the group can probably discuss three questions at the depth one 
looks for in a focus group. Displaying answers to the questions not discussed satisfies 
the group's curiosity about how others responded. Second, the displayed cards aid 
discussion. 



The facilitator can ask to hear from those who responded yellow before those who 
responded blue, or ask someone to explain a yellow response if only those who 
responded blue have spoken. And participants know when they explain their responses 
whether they are in the majority or minority and gauge the detail of their explanations 
accordingly. 

Going into the meeting, the facilitator will have an idea about which three questions he 
would most like to have the group discuss — that is, on which three questions he most 
wants to collect data. We suggest that the facilitator maintain some flexibility here to 
allow participants to add or substitute questions that are most compelling for them, or 
to discuss a question with a particularly interesting response pattern. 

The facilitator should explain at the beginning of the meeting, and again as 
participants begin to complete this questionnaire, that these instruments are meant to 
start discussion and the "real data" are the group's discussions. Still, a participant may 
take exception to the wording of a statement on the questionnaire or find it impossible 
to choose between the blue and yellow statements in a specific pair. The facilitator 
should take this opportunity to reinforce that it is the discussion that is important, and 
ask the participant what it is in his or her experience that makes this statement or this 
choice difficult. 

Because the data do lie in the discussion, it is vital to capture a detailed record of this 
and subsequent discussions in the focus group. We recommend having a second 
researcher, who understands the purpose of the meeting and is neutral on these issues, 
take detailed notes. We also recommend having the notetaker and facilitator take time 
immediately after the meeting to fill in the notes and identify major themes of the 
discussion. Alternatives include tape recording or having a single researcher facilitate 
discussion and at the same time capture key points on flipchart pages. 

We have pointed out that the second page of this questionnaire was collected but not 
discussed in our focus groups. In each of the states in which we conducted this study, 
the state department of education chose to use this opportunity to gather information 
about the status of specific initiatives. In one state this additional information provided 
a context that allowed state staff to better understand the data in the discussions. In 
another, it provided an opportunity for feedback on some specific policy decisions the 
state education department was about to make. In a third state, it allowed us to collect 
data for two inquiries at one meeting. These are all reasonable accommodations 
between the researcher and client. When considering adding survey-only questions, 
consider three things. First, how much time will the additional questions take away 
from focus group discussion? Second, are the additional survey questions compatible 
with the discussion questions, i.e., do they ask participants to think about details that 
may inform their responses to the discussion questions, or do they set up a competing 
agenda? Third, does the client understand the limitations of such a small sample? 

We should note that the content of this questionnaire deals broadly with school reform 
because the inquiry in New York state was quite broad. The same formats can be used 
for a more narrowly-focused inquiry by narrowly focusing the questions asked. 
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Focus Group tool 1 



Understandings and implementation of reforms 



BLUE check YELLOW 

one 


1 "Implementing high standards and 
raising the achievement of all 
students" is a good description of 
where our change efforts are 
focused. 






1 Our local needs and efforts are 
focused elsewhere. 


2 We are incorporating elements of 
the standards frameworks into 
existing teaching practice (e.g., 
cross referencing lessons to 
standards or adding activities). 






2 We are rethinking our teaching 
assumptions and strategies, using 
the standards frameworks as a 
foundation to make substantial 
reforms in practice. 


3 We have needs assessment and 
planning processes in place that 
make it possible for us to identify 
and achieve the major changes we 
need to make in the learning 
environment. 






3 Our needs assessment and 
planning processes seem to 
consume a lot of energy without 
leading to substantial improvements 
in our learning environment. 


4 We use state and local student 
assessment results to inform 
curriculum revisions, professional 
development, and community 
dialogue. 






4 Student assessments are not 

particularly helpful to us for revising 
curriculum, planning professional 
development, or community dialog. 


5 It is difficult for our school (district) 
to make progress because of 
competing convictions about the 
direction we should take. 






5 We know where we need to go as 
a school and district and support 
each other on that journey. 


6 My district would require a new set 
of skills and/or a new base of 
knowledge to achieve a high- 
standards learning environment for 
every child. 






6 My district has the skills and 

knowledge necessary to achieve a 
high-standards learning 
environment for every child; all we 
need is the opportunity and 
support. 
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Focus Group tool i continued 



Understandings and implementation of reforms 



7 For each of the initiatives below, indicate if you are knowledgeable about it 

personally, if your school (or district) is involved in or implementing it, and if it is 
central to your school's (or district's) reform efforts. An X indicates "yes", a blank 
indicates "no". 



perso nally 
knowledgeable 



we are involved/ central to 
implementing our reforms 



Commissioner's Regulation 100.11, [ 



shared decision making 

Developing standards-based [ 

curriculum 

An All-Regents program [ 

New assessments [ 

Schoolwide program [ 

Early literacy programming [ 

Universal pre-kindergarten [ 

New diploma requirements [ 



] [ ] [ ] 

] [ ] [ ] 

] [ ] [ ] 

] [ ] [ ] 

] [ ] [ ] 

] [ ] [ ] 

] [ ] [ ] 

] [ ] [ ] 



8 What is the one thing your school (or district or community) is doing that is most 
powerful for helping you become the kind of school(s) you want to be? (This does 
not have to be anything on the list above or anything associated with the State 
Education Department.) 
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State roles and strategies in scalbig up reform: dot voting 

In much of the discussion in these focus groups we asked about things — such as the 
status of reforms and the support system — from which we infer the reach and effects 
of state department of education strategies. It has also been important for the purposes 
of this study to ask focus group participants directly what approaches they have seen at 
the state level. 

We used the following instrument, without modifying wording or format, with focus 
groups in all three of the states in which we conducted this study. We used it both 
with practitioner groups and with support system groups. Because this exercise created 
a common data set across groups, we were able to compare responses across states, 
initiatives, and/or groups. 

The scales on this instrument come from the inquiry framework (see page 1) 
developed with state department of education representatives. The questions that can 
be framed in comparisons such as "Has the state department chosen to act in manner 
A or manner B?" are represented as scales here. Note that the participant not only 
identifies one of two competing strategies as dominant (the left or right side of the 
scale) but also rates on a 1 to 5 continuum how strongly (or consistently) the state 
department acts in that way. 

We prepare the room for this exercise before the meeting by posting eight large posters 
around the wall, with each poster containing one scale. Each participant has at his seat 
eight large adhesive dots (available at office supply stores). Participants are asked to 
think about the initiative that has been the topic of the meeting and to identify the state 
department of education's roles and strategies within that initiative. Responses are shared 
by silent dot voting; that is, participants indicate their responses to each question by 
placing a dot on the appropriate place on the scale. Each participant places one dot on 
each of the eight posters. To help participants organize their "votes" before going to the 
posters on the wall, we hand out the single sheet included here on page 10. 

When voting is completed, we begin the discussion by asking which of the posters 
interests or intrigues participants. This is usually one on which there is a strongly split 
opinion. As with the earlier discussion, this is an opportunity for participants to share 
their different experiences and frames of reference. And as with the earlier discussion, 
there is not time to discuss all eight scales. The researchers collect the posters at the 
end of the day so that the surface-level responses to all questions are captured. 

We placed this activity in the center of our three-hour meetings for several reasons. It 
gets people up from their chairs and moving around in the middle of the meeting, and 
so combines well with a break. It is also a mental change of pace, a different kind of 
conversation starter between two that are very much alike. This discussion also allows 
people to air their observations and opinions about state department of education 
activities so that, when we move on in the next conversation to the larger support 
system, that time need not all be taken up talking about the state department. 

In a variation on this activity, participants in Vermont were given eight each of two 
different colors of dots. These groups, which were discussing standards and assessment 
initiatives, voted with one color for state strategies regarding standards frameworks 
and another color for those regarding assessments. The discussion that followed dealt 
largely with why different strategies seem to have been adopted in different initiatives. 
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Focus Group tool 2 



State roles and strategies in scaling up reform 

In scaling up these reforms statewide, the State Education Department has . . . 



1 1 1 1 1 
5 4 3 2 1 

acted in a rule-making 
and monitoring role 


l 

0 


1 

1 


1 1 1 1 

2 3 4 5 

acted in an assistance 
and capacity-building role 


1 1 1 1 1 
5 4 3 2 1 

advocated a specific 
approach by schools 


1 

0 


1 

1 


1 1 1 1 

2 3 4 5 

supported multiple 
approaches by schools 


1 1 1 1 1 

5 4 3 2 1 

set a modest pace, 

stages 


1 

0 


1 

1 


1 1 1 1 

2 3 4 5 

aimed for full implementation, 
all at once 


l l l l l 

5 4 3 2 1 

expected implementation 
and use everywhere 


1 

0 


1 

1 


1 1 1 1 

2 3 4 5 

expected implementation 
and use in pockets 


l l l l l 

5 4 3 2 1 

used authoritarian processes 


1 

0 


1 

1 


1 1 1 1 

2 3 4 5 

used democratic processes 


1 1 1 1 1 
5 4 3 2 1 

provided incentives 
and rewards 


1 

0 


1 

1 


1 l 1 1 

2 3 4 5 

provided corrections 

and penalties 


1 1 1 1 1 
5 4 3 2 1 

expected local adaptations 


1 

0 


1 

1 


1 1 1 1 

2 3 4 5 

expected uniformity 



54321 012345 

focused on integrating these dealt with these 

initiatives with other goals initiatives in isolation 
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Support systems for reform: participant questionnaire 

At this point in our practitioner focus groups, practitioners will have talked about their 
experiences with the initiative(s) under discussion and the strategies used by the state 
education department in advancing that initiative. Our third discussion is about the 
nature and quality of the supports available to and used by schools for the initiative. 
One of the premises of our study is that the partnerships the state department of 
education forms, the intermediaries it uses to carry its messages, and the new initiative- 
specific networks and structures it creates are central features of the new strategies. 

The conversation-starting exercise for this discussion is in the format used for the first 
discussion. Once again participants have five minutes to complete the two-page 
questionnaire and 40 minutes to discuss approximately three of the questions on the 
first page. Once again the second page is not discussed. Rather, it is used to help 
collect specific information about which the state education department is 
interested — in this case about which supports are useful. The questionnaire is 
collected. 

Looking at this instrument offers an opportunity to reflect further on the nature of a 
focus group interview. If the main objective of the meeting were to collect the kind of 
information on the second page, it would be more efficient to mail a survey. One 
could ask many more questions of this "yes or no" variety and could ask them of many 
more people. A researcher will choose a focus group over a survey when the questions 
are more complex, when the possible range of answers is not known, and/or when it is 
valuable to have respondents compare experiences. To capitalize on this opportunity 
the discussion time should move as quickly as possible beyond "yes or no" and into 
"why,” "how,” "under what circumstances,” and "with what result." 

These are questions that will lead where the group takes them, and not everyone will 
engage in every part of the conversation. If, in addition to this rich conversation, the 
researcher or client wants to be sure to leave the meeting having learned everyone's 
answer to a specific closed-ended question such as "Do you use the NYSED Web 
site?" an efficient way to obtain those answers is with a brief survey embedded in the 
meeting. The second page of this instrument is such a brief survey and meets the 
criteria discussed earlier: completing it takes very little time and it reminds participants 
of features of the support system, possibly enriching their discussion. 
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Focus Group Tool 3 



Support systems for reform 



BLUE check YELLOW 

one 


1 The messages our school (or 
district) receives about where our 
reforms should be going and how 
to get there are consistent across 
sources. 






1 We hear so many different 
messages about what reform 
consists of that there is a lack of 
focus. 


2 The support system necessary for 
successful standards-based 
reform is not yet in place. 






2 The supports available to teachers, 
schools, and communities from 
various sources is sufficient for 
standards-based reform of 
instruction and assessment. 


3 The people whose behaviors must 
be supported or modified in order 
for reform to succeed are 
appropriately targeted by 
communication and assistance 
strategies. 






3 The people who ultimately must 
make changes if reforms are to 
succeed are not receiving needed 
information and support. 


4 Supports are focused on specific 
knowledge acquisition or 
procedural change by individuals, 
and so are not creating conditions 
for self-renewing reform in schools 
or communities. 






4 The supports being provided to 
teachers, schools, and/or 
communities are creating self- 
sustaining change processes that 
are likely to continue after the 
supports are gone. 


5 In leading standards-based 
instructional and assessment 
reforms, the State Education 
Department is responding to needs 
expressed in the schools and 
communities of the state. 






5 The State Education Department is 
creating a new dialogue about 
standards-based reform, striving to 
persuade schools and communities 
of the need for such reform. 




20 



12 





Focus Group tool 3 continued 

Support systems for reform 



6 For each of the following sources of information and support, please indicate in the 
first column which you rely on most to get clear information about what you 
should be doing in your reforms. Then indicate in the second column which you 
rely on for help in getting there. 



get sense of 



Printed policies and guidance from the 

State Education Department [ ] 

Printed guidance from other sources 

(example: ) [ ] 

Professional development/training events 

offered by the State Education [ ] 

Professional development/training events 
offered by other organizations 

(example: ) [ ] 

State Education Department public forums [ ] 

State Education Department staff [ ] 

School district central office staff [ ] 

Staff within my and other [ ] 

BOCES staff [ ] 

Teacher Resource Centers [ ] 

Professional membership organizations 

(AS CD, NYSUT, NYSACE, etc.) [ ] 

State-sponsored technical assistance centers 
(BETAC, SETRC, Sharing Success, etc [ ] 

Published research [ ] 

Internet, including NYSED Web site [ ] 

Federal technical assistance centers 

(NYTAC, NERRC, Eisenhower Alliance, etc.) [ ] 

Colleges and universities [ ] 

Private consultants [ ] 

Other: [ ] 



get implementation 



[ ] 
[ ] 
[ ] 

[ ] 
[ ] 
[ ] 
[ ] 
[ ] 
[ ] 
[ ] 

[ ] 

[ ] 
[ ] 
[ ] 

[ ] 
[ ] 
[ ] 
[ ] 




21 



13 



"The last word" 



We often end a focus group with a brief exercise we have come to call “the last word.” 
The example from our New York state practitioner focus groups follows on page 15. 
Most often the answers to these one to three questions are not discussed, but they can 
be. This exercise can be done quickly and allows each participant a final opportunity to 
be heard by the researcher and the client. The question(s) asked should cover the scope 
of the entire meeting's conversations. Responses are anonymous, collected, and 
transcribed without modification as part of the focus group data. 

Several things are accomplished by "the last word." A participant who has not spoken a 
great deal in the discussions gets equal time in this exercise. A participant who has a 
strong point to make, even if he or she has made them in discussion, can be sure that in 
this exercise those points will not be “massaged,” or combined or translated by the 
researcher. A participant who has a more articulate or detailed suggestion to make as a 
result of the full day's discussions than he had earlier in the day can get that suggestion 
on the record. A well-chosen question can bring back to the main point of the meeting 
a set of discussions that has ranged far. 

We found in our study that the transcription of these last participant statements was 
often what state department of education partners wanted to read first, the “bottom 
line” feedback from the field. 
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Focus Group Tool 4 



The last word 



Based on the State Education Department actions you have observed (not necessarily 
the written policies), what are the major messages you get from the State about high 
standards and/or the accompanying new assessments? 



What are the most common misunderstandings among practitioners about standards 
and/or assessment in New York? 



If you could tell the State Education Department one thing to do to improve 
understanding and implementation in schools of high standards and new assessments, 
what would it be? 
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Further adaptations: support system questionnaire 

Our belief that the formats presented on pages 7-15 can be modified for many 
research questions and many groups is based on having used such modifications. 

Our research design required that we collect different data from groups of support 
providers than from groups of practitioners. Following on pages 17-19 is one 
questionnaire used in the support system focus groups in Vermont. The basic format 
is the same one used twice in the New York practitioner groups: color-coded paired 
statements, the answers to which participants display prior to discussion. The content 
is quite different and is based in part on the findings from the Vermont practitioner 
focus groups. 
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Focus Group tool 5 



Implementation and use of standards and assessment framework 



BLUE check YELLOW 

one 


1 Most of the important audiences for 
school reform (teachers, 
administrators, students, parents, 
community) understand essential 
information about Vermont's 
standards and assessment 
frameworks. 






1 People in schools are just beginning 
to understand essential information 
about Vermont's standards and 
assessment frameworks, and other 
important audiences are not yet 
informed. 


2 Most schools are using processes 
for standards and assessment 
reform that they have used for 
other initiatives, and therefore have 
the needed skills and expertise. 






2 Most schools are developing new 
processes and having to acquire 
new skills to implement standards 
and assessment reform. 


3 Implementing and using the 
standards and assessments are 
consistently more difficult for 
schools or staffs with certain 
characteristics (e.g., small schools, 
secondary level teachers) than for 
others. 






3 Differences in difficulty 
implementing and using the 
standards and assessments are 
due largely to individual 
circumstances, and are not part of 
any broader pattern. 


4 Most schools are incorporating 
elements of the standards into 
existing teaching approaches (e.g., 
cross-referencing lessons to 
standards or adding activities). 






4 Most schools are rethinking their 
teaching assumptions and 
strategies, using the standards 
framework as a foundation to make 
substantial reforms in practice. 


5 Most schools are making use of 
state and local assessment results 
to inform curriculum revisions, 
professional development, and 
community dialogue. 






5 Most schools are not relying on 
student assessment data to inform 
their improvement decisions. 


6 The standards movement is the 
best hope for combining excellence 
with equity for all students, raising 
the bar for everyone. 






6 The standards movement results in 
benefits for some students at the 
expense of others. 



ERIC 
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Focus Group tool 6 



Specific actions and approaches to advance standards and assessment reform 



The following are suggestions by practitioner focus groups (teachers and administrators) about 
specific actions and approaches the State Department of Education could take next to advance 
standards and assessment reform. Indicate with which of these ideas you agree. Among those with 
which you agree, indicate up to four which you feel are vital to the success of these reforms. 



Agree 

1 Put effort into reinforcing publicly that this is the direction of education 
reform in Vermont for the long haul. 



2 Provide reform implementation status reports for the state as a whole. [ ] 

3 Provide examples of successful implementation strategies of individual [ ] 

teachers, schools, and districts. 

4 Promote positive coverage by the media of progress being made, [ ] 

Vermont’s leadership role in standards and assessment, and the 
professionalism of Vermont’s teachers. 

5 Engage institutions of higher education in a more active discussion of [ ] 

using the standards in teaching undergraduates and in preparing 

teachers. 

6 Clarify the role and purpose of the standards in grades 11 and 12, after [ ] 

the grade 10 assessments. 

7 Articulate expectations (e.g., with benchmarks) for grades 5-6, to [ ] 

facilitate transitions to grade 7. 



8 Provide centralized access to and/or information about locally developed [ ] 

assessments aligned with the standards. 

9 Provide centralized access to and/or information about locally developed [ ] 

units and courses of study aligned with the standards. 

10 Provide guidance about commercial products (curriculum, texts, tests, [ ] 

programs) aligned with the standards. 

11 Provide user-friendly versions or ’’interpretations” of the standards [ ] 

frameworks to make them approachable for more audiences. 

12 Provide school level interpretations of New Standards Reference Exam [ ] 

results that are explicitly linked to Vermont standards. 

13 Designate or create a dedicated point of contact in each district, beyond [ ] 

the Superintendent, to receive and disseminate information from the 

State about standards and assessments. 

14 Articulate and reinforce appropriate roles in standards and assessment [ ] 

reform for formal leaders (principals, superintendents). 



Vital 



[ ] 
[ ] 

[ ] 

[ ] 

[ ] 
[ ] 
[ ] 
[ ] 
[ ] 
[ ] 
[ ] 
[ ] 

[ ] 
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FOCUS GROUP TOOL 6 continued 



Specific actions and approaches to advance standards and assessment reform 

\ 



Other vital next steps? 
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